Skip to content

branch-3.1: [feature](maxcompute)support maxcompte catalog read project-schema-table. #57012 #57286

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Oct 30, 2025
Merged

branch-3.1: [feature](maxcompute)support maxcompte catalog read project-schema-table. #57012 #57286

merged 1 commit into from
Oct 30, 2025

Conversation

@github-actions
Copy link
Contributor

Cherry-picked from #57012

@hubgeter
[feature](maxcompute)support maxcompte catalog read project-schema-ta…
67561b7 
…ble. (#57012)

### What problem does this PR solve?
Problem Summary:
In the previous MaxCompute architecture, and its mapping in Doris, the
hierarchy was: `project / database -> table / table`. When creating a
catalog, users needed to specify the property `mc.default.project`,
which indicated the default project to access. In this structure,
executing `SHOW DATABASES` would list other projects.

After MaxCompute introduced the concept of schemas, the hierarchy
changed to: `project / catalog -> schema / database -> table / table`.
Here, the project is at a higher level, and `SHOW DATABASES` should now
list all schemas under the current project. As a result, users need to
create a separate catalog for each project, specifying a different
`mc.default.project` property.
doc:
https://help.aliyun.com/zh/maxcompute/user-guide/schema-related-operations?spm=a2c4g.11186623.help-menu-search-27797.d_0#9d8326491bagj

To maintain compatibility with the old version, a catalog property
`mc.project.schema.table` is introduced:
    - When the property is true, the new architecture is used.
    - When the property is false, the old architecture is used.
@github-actions github-actions bot requested a review from morrySnow as a code owner October 24, 2025 02:48
@Thearas
Copy link
Contributor

Thearas commented Oct 24, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Oct 24, 2025
@Thearas
Copy link
Contributor

Thearas commented Oct 24, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32417 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 67561b71a108f1d7a5ac7d895f108292297470ec, data reload: false

------ Round 1 ----------------------------------
q1	17610	5424	5484	5424
q2	2018	395	283	283
q3	12069	1211	734	734
q4	10312	866	451	451
q5	9168	2357	2105	2105
q6	184	162	132	132
q7	881	741	604	604
q8	9325	1454	1141	1141
q9	5260	4961	4912	4912
q10	6770	2285	1806	1806
q11	475	275	255	255
q12	334	354	213	213
q13	17790	3585	3010	3010
q14	220	231	224	224
q15	549	470	456	456
q16	424	420	368	368
q17	581	851	356	356
q18	6886	6430	6324	6324
q19	1220	957	517	517
q20	315	329	201	201
q21	2760	2176	1908	1908
q22	1032	1029	993	993
Total cold run time: 106183 ms
Total hot run time: 32417 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5552	5522	5557	5522
q2	237	330	234	234
q3	2245	2628	2423	2423
q4	1412	1792	1368	1368
q5	4558	5091	5056	5056
q6	171	172	135	135
q7	2137	2045	1851	1851
q8	2677	2886	2739	2739
q9	7459	7442	7329	7329
q10	3116	3314	2806	2806
q11	596	550	508	508
q12	685	790	645	645
q13	3490	3911	3334	3334
q14	293	291	283	283
q15	527	469	492	469
q16	442	477	440	440
q17	1225	1723	1263	1263
q18	7657	7513	7409	7409
q19	782	979	1131	979
q20	2039	2094	1908	1908
q21	5399	4792	4551	4551
q22	1133	1075	1037	1037
Total cold run time: 53832 ms
Total hot run time: 52289 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193004 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 67561b71a108f1d7a5ac7d895f108292297470ec, data reload: false

query1	964	397	396	396
query2	6177	1918	1860	1860
query3	8689	207	196	196
query4	33585	24070	23951	23951
query5	4305	599	471	471
query6	310	202	172	172
query7	4217	481	316	316
query8	303	241	228	228
query9	9499	2577	2578	2577
query10	490	320	258	258
query11	18155	15498	15215	15215
query12	154	112	116	112
query13	1584	534	409	409
query14	9158	6638	7423	6638
query15	235	192	175	175
query16	8120	673	506	506
query17	1541	792	613	613
query18	2154	416	323	323
query19	225	212	168	168
query20	140	121	125	121
query21	229	137	128	128
query22	4580	4524	4368	4368
query23	35842	34513	34281	34281
query24	7254	2688	2709	2688
query25	527	502	436	436
query26	794	286	175	175
query27	2149	473	363	363
query28	4824	2194	2157	2157
query29	712	634	473	473
query30	243	198	158	158
query31	1014	902	846	846
query32	87	65	59	59
query33	550	411	306	306
query34	753	852	521	521
query35	796	842	725	725
query36	1040	1078	960	960
query37	109	92	74	74
query38	4074	3996	4027	3996
query39	1536	1497	1495	1495
query40	215	121	106	106
query41	49	49	50	49
query42	120	101	108	101
query43	508	509	491	491
query44	1327	844	827	827
query45	203	178	179	178
query46	873	1043	675	675
query47	2011	2015	1932	1932
query48	409	419	335	335
query49	752	495	402	402
query50	662	721	423	423
query51	7233	7417	7200	7200
query52	104	105	93	93
query53	237	262	192	192
query54	544	536	482	482
query55	79	85	83	83
query56	295	305	252	252
query57	1276	1256	1222	1222
query58	252	218	216	216
query59	3122	3154	3070	3070
query60	301	286	280	280
query61	129	110	154	110
query62	820	765	704	704
query63	242	199	192	192
query64	3703	1010	644	644
query65	3427	3310	3335	3310
query66	1066	409	339	339
query67	16572	15828	15711	15711
query68	8393	804	527	527
query69	500	308	263	263
query70	1194	1129	1123	1123
query71	424	299	268	268
query72	5139	3842	3821	3821
query73	651	743	348	348
query74	10318	9357	9308	9308
query75	4369	3235	2658	2658
query76	3779	1185	756	756
query77	766	356	266	266
query78	10362	10493	9636	9636
query79	5129	869	585	585
query80	639	517	453	453
query81	487	253	218	218
query82	227	119	88	88
query83	169	155	141	141
query84	290	101	78	78
query85	770	375	303	303
query86	341	294	299	294
query87	4291	4350	4240	4240
query88	4329	2406	2400	2400
query89	431	330	288	288
query90	2128	196	191	191
query91	136	136	109	109
query92	68	55	53	53
query93	3615	874	528	528
query94	657	410	305	305
query95	353	280	267	267
query96	486	603	282	282
query97	3175	3260	3148	3148
query98	224	204	204	204
query99	1446	1393	1322	1322
Total cold run time: 298935 ms
Total hot run time: 193004 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.43 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 67561b71a108f1d7a5ac7d895f108292297470ec, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.04	0.03
query3	0.22	0.07	0.06
query4	1.63	0.11	0.11
query5	0.51	0.52	0.52
query6	1.13	0.74	0.72
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.57	0.49	0.50
query10	0.56	0.54	0.55
query11	0.15	0.11	0.10
query12	0.14	0.11	0.11
query13	0.63	0.60	0.60
query14	0.79	0.80	0.82
query15	0.85	0.84	0.83
query16	0.38	0.38	0.38
query17	1.07	1.05	1.05
query18	0.24	0.22	0.22
query19	1.88	1.84	1.91
query20	0.02	0.01	0.01
query21	15.37	0.91	0.59
query22	0.74	0.90	0.56
query23	15.10	1.43	0.58
query24	2.93	1.40	1.50
query25	0.15	0.11	0.07
query26	0.25	0.15	0.14
query27	0.05	0.06	0.06
query28	13.94	1.05	0.43
query29	12.56	3.89	3.29
query30	0.26	0.09	0.06
query31	2.83	0.62	0.38
query32	3.24	0.54	0.46
query33	3.04	3.02	3.04
query34	16.55	5.19	4.55
query35	4.59	4.56	4.61
query36	0.66	0.50	0.51
query37	0.08	0.06	0.05
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.02	0.03
Total cold run time: 103.67 s
Total hot run time: 29.43 s

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 0.00% (0/100) 🎉
Increment coverage report
Complete coverage report

@morrySnow morrySnow merged commit ddaf8ab into branch-3.1 Oct 30, 2025
23 checks passed
@github-actions github-actions bot deleted the auto-pick-57012-branch-3.1 branch October 30, 2025 06:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@Thearas @doris-robot @hello-stephen @morrySnow @dataroaring @hubgeter